The MicroGrid: Using Emulation to Predict Application Performance in Diverse Grid Network Environments

Improvements in networking and middleware technology are enabling large-scale grids that aggregate resources over wide-area networks to support applications at unprecedented levels of scale and performance. Unfortunately, existing middleware and tools can not in general inform a user as to the suitability of a given grid topology for a specific grid application. Instead, users generally develop ad hoc performance models for mapping their applications to resource and network topologies. Because application behavior alone is complex, and resource and network behavior further complicate the picture, users are typically reduced to nearly blind experimentation to discover how to deploy their application in the new grid environment. Only after the fact can a user discovers if the match was a good one. Further, even after finding a desirable configuration, there is no basis on which to determine if a much better configuration exists. We present a richer methodology for evaluating grid software and diverse grid environments based on the MicroGrid grid emulator. With the MicroGrid, users, grid researchers, or grid operators can define and emulate arbitrary collections of resources and networks. This allows study of an existing grid testbed under controlled conditions or even to study the efficacy of higher performance environments than are available today. Further, the MicroGrid supports direct execution of grid applications unchanged. These application can be written with MPI, C, C++, Perl, and/or Python and use the Globus middleware. This enables detailed and accurate study of application behavior. 0This material is based upon work supported by the National Science Foundation under Grant #9975020 and #0103759. This paper presents: (1) the first validation of the MicroGrid for studying whole-program performance of MPI Grid applications and (2) a demonstration of the MicroGrid as a tool for predicting the performance of applications on a range of grid resources and novel network topologies.